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WHAT IS CLAIMED IS: 



1. 



system for maintaining a background model of an image 



sequence having a pKjrality of pixels, comprising: 

a pixel proc^sing module that processes the image sequence on a 
pixel scale; and 

at least one refinement module that processes the image sequence 
on a spatial scale other than the pixel scale. 



The system of claim 1, wherein the pixel processing module further 



^/compris^determining an initial tJackground model and providing an initial pixel 
assignment to each of the plurality of pixels. 



3. The System of claim 1 , wherein a first refinement module is a region 
1 5 processing module th^process^sihe image sequence on a regional scale. 



20 



4. The system of daim 3, wherein the region processing module 
further comprises considering a re^fionship between at least some of the 
plurality of pixels to provide pixel assignment. 

5. The system of claim ^whereih^a second refinement module is a 
frame processing module that processes the imtage sequence on a frame scale. 
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The system of claim 5, wherein the frame processing module 
further determines a background model that most accurately represents an 
actual background of the image sequence and performs one of: (a) retaining a 
current background model; (b) substituting a more accurate background model 
in place of Wie current background model. 



7. the system of claim 5, further comprising a postprocessing module 
that provides enhancement of thejnlagesequence. 

1 0 8. The s\stem of claim 7, wherein the enhancement is speckle 

removal. 



9. The system V claim 7, wherein the postprocessing module 
provides enhancement after\he pixel pressing module and before the frame 
15 processing module. ^ ' 
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10. The system of claim \ wherein the postprocessing module provide 
enhancement after the frame proc^s^ng module and before the region 
processing module. 



A 

modules, comprising: 



computer-readable medium having computer-executable 



y 
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a pixel proces^pg module that processes the image sequence on a 
pixel scale; and 

at least one refinement module that processes the image sequence 
on a spatial scale other than the Wei scale. 

The apparatus of claim 1 l^jwherein the refinement module 
processed the image sequence on scale larger than the pixel scale. 

13. uie apparatus of claim 12^vh.erein a first refinement module is a 
10 region processing module that processes the image sequence on a region scale. 
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14. The apf^aratus of claim 13^jwhe^ein a second refinement module is 
a frame processing module that processes the image sequence on a frame 
scale. 

15. The apparatus oAHaim 14, further comprising a postprocessing 
module that provides enhancemerk of t^lmage sequence. 



"^16. A meffitJdsfor maintaining a background model of an image 
20 sequence having a plurality erf pixels, comprising: 

processing the image sequence on a pixel scale so as to determine 
a current background model ar d provide an initial assignment for each of the 



plurality of pixels; and 



/ 
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refining the pixel processing by processing on a spatial scale other 
that the pixel scale to further refine at least one of: (a) the current background 
model; (b) the initial pixel assignments. 

5 ifT^* The method of claim 16^/yherein refining further comprises 

providing avregion processing mocfule that processes the image sequence on a 
region scale. 

\ 

1 8. Th^method of claim 1 7^wterein refining further comprises 
10 providing a frame pressing module that processes the image sequence on a 

frame scale. \^ 

\ 

19. The method \fclaim 18, wherein refining further comprises 
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providing a postprocessing module that enhances the image sequence. 



20. The method of claimVI9, wherein the postprocessing module 
enhances the image sequence by prbvidiil^speckle removal. 



Zlv A system for processing an image sequence having a plurality of 
20 frames, each w^me cogt^ining ja pixel, comprising: 

'corft^jtation nrtodule that provides previous values of tl^epixel; 
la pipcHo^oXmodule that predicts a predicted future value for the 
pixel using the previOL^pl^gl values; and 
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a declaration module that classifies the pixel by comparing the 



predicted future va 



ue to an actual future value of the pixel. 



22. The system of claim 21 , wherein the computation module provides 
5 previous predicted pixel values and previous actual pixel values. 



23. The systerr^ of claim 22, wherein the prediction module further 
comprises: 

an actual prejjkffibp 'module that predicts the predicted future value 
1 0 using the previous actual pixtel fokjef ; and 

a prepctedp^flr^tion module^fiat predicts the predicted future 
value using the preyioy^pF^dici^d pixel yalues. 

24. The system of clajm^3 v further comprising an update module that 
1 5 determines predicted futupef^ value'having least amount of error and appends this 

predicted future valw^to the previousWedicted pixel values. 

\ 

25. The system of claim 21 , wherein: 

the prediction moduleijses \ prediction technique having prediction 
20 parameters to predict the predicted future parameters. 
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26. The system of claim 25, further comprising a recomputation module 
that recomputes at least some of the prediction parameters to form new 
predictionparameters. «^ 

5 27. Vhe system of claim 26, wherein: 

tn^ declaration module^classifies the pixel as either a background 
or a foreground p\cel; and 

the re^mputation module is capable of replacing the prediction 
parameters with the n^w pregicJkJn parameters. 
10 / \ 

28. A comp^r-r^adable me/fium having computer-executable 
modules, comprising 

a computation module that supplies previous predicted values of a 
pixel contained in an the previous predicted values comprising previous 

1 5 actual values and pr^ious predicteckvalues of the pixel; 

addiction module that Wdicts at least: (a) a predicted future 
value for the pixel using the previous predicted pixel values; and (b) a predicted 
future value for the pixel using the previous actual pixel values; and 

a declaration module that classifies the pixel by comparing at least 
20 one of the predicted future values to an actual Viture value of the pixel. 



54 



19. The apparatus of claim 28, further comprising an update module 
that deiides which of the predicted future values has the least error and includes 
this predated future value to the previous^redicted pixel values. 

5 30. \ The apparatus of claim 28, wherein the prediction module uses a 

prediction technique having prediction values to predict the predicted future 
values. 

31 . The apparatus of claim 30, further comprising a recomputation 
1 0 module that recalculates at least some of the prediction parameters to form new 

prediction paranrtetersA j 

32. Th^ afafaratufe of claifli 30, wherein the prediction parameters are 
recalculated usiKg the actualWure vajye^and the previous actual values of the 

15 pixel. 
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33. The apparatus of clain^32, wherein: 

the declaration module classifies the pixel as either a background 
or a foreground pixel; and 

the recomputation module is\capable of replacing the prediction 
parameters with the new prediction paramett 
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5 values; 



10 



15 
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3ty. A method of processing an image sequence having a plurality of 
frames, e^ch frame containing a pixel, comprising: 
storing previous values of the pixel; 

predicting at least two future values of the^ixeTusing the previous 



determining an actual value of the pixel; and 
comparing the actual pixel value with at least one of the future pixel 



\ 

values so as to classify the pixel 



35. The me; 



both of: (a) actual previous pixe\ values; ana (b) predicted previous pixel values. 




'^in storing further comprises storing 



36. The me N 



an actual future pixel value; (b) a 



erein future pixel values comprise: (a) 



future pixel value. 



37. The metjadd of claim 36, wherein 

value^N 



actual future pixel value is^alculated from the actual previous 



pixel values; and 



\ 



. \ 



the predicted future pixel value is calculated from the predicted 
previous pixel values. 
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318. 



The method of claim 34, wherein comparing further comprises 



comparing the actual pixel value to both the actual future pixel value and the 
predicted future pixel value. 



5 39. ftie method of claim 36, further comprising finding the future pixel 

value closest in v^lue to the actual pixel value and adding this future pixel value 

\ ^ 

to the predicted previous pixel valu£s^ 
\ 

\ 

\ 

40. A systemNfor processing an image having pixels, comprising: 
10 an intersectlpp^modjjle that groups the pixels to form an 

intersection image based o)^ an imi^rtdass^ of each pixel; 



a histogram msfqule that determines a property of the pixels within 

\ iK * / 

the intersection image to\:reate N a boundary of a foreground region; and 

\\\/ 

a backprojectioh^jbjdule that fills in the foreground region within the 

1 5 boundary to segment the foreground, region from the image. 

\ 

\ 

s 

41 . The system of claim 40, fuller comprising an image differencing 

module that identifies moving pixels u$ing ^plurality of previous images earlier in 

S \ 

time than the image. \ 

A 

42. The system of claim 41 , wherein: \ 

y \ 

the image differencing module takes th\? difference of the plurality 

\ 

of previous images that are adjacent in time to form a crifferenced image; and 

\ 
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the intersection module uses the differenced image to form the 
intersection ima*g6 that contains moving foreground pixels. 

43. The system of claim 42, wherein the intersection module forms the 
5 intersection image by ralculatingJhe„intersection of the differenced image and a 
foreground image containing pixels classified as foreground pixels. 



10 
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44. The system ofalaim 42^\A(herein the histogram module calculates 
the property of the pixels in tne intersection image. 

45. The system o^dal^n H4, wherein the property of the pixels in the 
intersection image are c^culatequsingiJix^l property values from a previously 
processed frame. 



46. The syste 



w 



)laim 4& wherein the property of the pixels is at 



least one of: (a) pixel color; (b) pixeUntensity; (c) pixel depth 



47. The system^f claim 45, wher^jn a frequency threshold is defined 

X \ 

such that pixels^pdjrring less than the frequency threshold are not included in 

/ \ 

20 the foreground region. \ 

\ 

48. The system of claim 44, wherein the\ackprojection module uses 
connectivity to determine which pixels to include within the boundary. 
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49. 

neighboring pix 



The system of claim 48, wherein connectivity includes finding 
Is with a similar property. 



I 

50. Thle system of claim 49, wherein the amount of connectivity is at 



\ 

least about four. \ 
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51. 

comprises: 



\ 

The sVstem of claim 49, wherein the backprojection module further 

\ 
\ 

entering ^jxels within the intersection image into a queue; 

\ 

comparing^ac^ pixel's histogram value with a histogram threshold; 



and 

includh 

value is greater than> 



/ 



the 



)bcel within the foreground region if the histogram 
stooram threshold. 



\ 



52. A computer-reajElabl^ medium having computer-executable 
modules, comprising: / \ 

an intersection module tkjat groups pixels in a current frame based 



on an initial classification of each pixel tb form an intersection image; 

a histogram module that de\^rmines a property of intersection 
image pixels to form a boundary of a foreground region; and 

a backprojection module that connects pixels within the boundary of 
the foreground region to divide the foreground region from the current frame. 
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>3. The apparatus of claim 52, further comprising an image differencing 
module Viat identifies moving pixeteusing a plurality of frames, each of the 
plurality oryframes being earlier in time that the current frame. 

54. \The apparatus of claim 53, wherein: 



the image differencing module calculates the difference between 

\ 



temporally adjacent frame to form a differenced image; and 

the intersection module uses the differenced image to group the 



10 pixels. 
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55. The appar£u}§ of qfaim^Twherein tme intersection module 

v / 

calculates the intersection o^jtfe differencejtima^e and a foreground image 
containing foreground pix^fe to^)rm the intersection image containing moving 
foreground pixels. 

\ 

56. The apparatus o^daim 5^ wherein the histogram module 

\ 

calculates a histogram fojxa property of^^h pixel within the intersection image. 
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57. The apparatus of claim 56, wherein the pixel property is at least 
one of: (a) pixel color; (b) pixel intensity; <cfpix\|^depth. 
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58. The apparatus of claim 56, wherein the backprojection module uses 
connectivity to find neighboring pixels with similar pixel properties. 



59. \ The apparatus of claim 58, wherein the connectivity is at least 
5 approximately four. 

\^ 

60. T{ie apparatus of claim 59, wherein the histogram value of each 

\ 

pixel within the intersection image is compared to a histogram threshold and 
included in the foreground region is thejjistogram value is greater than the 
1 0 histogram threshold. \ 

61. A method of proces^ing^fTimage containing pixels, comprising: 

finding arf interaction image withiri the image using an individual 

f\ 

pixel property; \ / A V 

1 5 determining a^quri^ry^f a foreground region using the 

intersection image; and 

filling in the fairground region within the boundary to segment the 
foreground region from/tne image. 



20 62. The method of claim 61 , whereirS determining a boundary further 

comprises calculating a histogram for each pixel in the intersection image. 
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>3. The method of claim 62, wherein filling in the foreground region 
further comprises backprojecting the histogram onto the boundary. 

64. \ The method of claim 63, wherein backprojecting further comprises 
5 using connectivity to determine which pixels to include within the foreground 
region. 



65. The method of claim 64, wherein using connectivity comprises 
comparing a histogram value of each pixel within the intersection image to a 
1 0 threshold and includiritt-the pixel wit hin th eforeground region is the histogram 
value is greater than the^togram threshold. 
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66. ThVmeTtiod o\clain/65, wherein the connectivity is at least 
about four. 



\ 



67. Asysiem for maintaining a background model of an image 

sequence containing a current frame, and previous frames, comprising: 

\ 

a stored model module that contains a plurality of stored 



background models; 



\ 



a background model module tXat uses the current frame to 

\ 

determine which of a current background moo^and the plurality of stored 
background models is the best background model. 
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58. The system of claim 67, wherein the background model 
module (determines the best background model by finding the background 
model with the lowest percentad^eofforeground pixels. 
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69. The system of claim 67, wherein the stored model module 
further comprises a frequency module that determines the frequency 
between testing \or replacement of tbe'plurality of stored background 
models. 



70. The 
plurality of stored 
number of frame 




71. Thesyst 



I of claim 69, wherein an update frequency of the 
nd^modLs is determined by a percentage of a 
ixel to regain stationarity. 

70, wherein the update frequency has a 



period of approximately 10 seco^s^^- 

\ 

\ 

72. The system of claim 69, yherein the frequency module 
determines whether there are enougk^ored background models. 

\ 

73. The system of claim 72, wherein N the frequency module selects 



a background model such that the plurality of background models is 



maximally diverse. 
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74; The system of claim 69, wherein the stored model module 
further comprises a benefit module that determines which of the plurality of 
stored background models shouldlaereplaced. 



75. 



le system of claim 74, wherein the benefit module replaces 
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the plurality of scored background models so as to maintain a maximally 
diverse set of stor^ background models. 

\ 

76. The system of claim 75, wherein the benefit module uses a 
clustering technique? to determine^wh^^of the plurality of stored 
background mod/ 



wherein the clustering technique is a 
close any two stored background 



77. The 5 
distance metric that determi 
models are to each 




78. A computer-readable mediuimhaving computer-executable 



\ 



modules, comprising: ^ 

a stored model module that contains a plurality of stored 
background models; and 
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a background model module that determines which of a 
current background model and the plurality of stored background models 
best represents\a background of a current frame. 

5 79. The Apparatus of claim 78, wherein the stored module 

comprises a frequency module tfiatdetermines an update frequency of the 
plurality of stored background models. 

80. The apparatus of claim 79, wherein the update frequency is 
10 determined by a percentage of a nurrjbeTbTframes required for a pixel to 

regain stationary 

81 . The apparatus of claim 80, wherein the update frequency is 
approximately 10 secor 



15 



82. The apparakis of claim 80, wherein the update frequency is 
approximately every 50 frames. 



83. The apparatus ^f claim 79^wl?erein the frequency module 
20 determines whether there are\nough stored background models. 
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84. The apparatus of claim 83, wherein the determination is made 
by the fi|equency module to provide a/maximally diverse set of stored 
background models. 
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85. \ The apparatus of claim 79, wherein the stored model module 

further comprises a benefit module that,determines which of the plurality of 

stored background models to replace. 
\ 



iete 



86. The apparatus of claim 85, wherein the determination is made 
10 by the benefit modVile to provide a maximally diverse set of stored 



background mo 




87. The apparatus p\ claim 86, wherein the benefit module 
determines which of the/fclurality of stored background models to replace 
based on a clustering techniq\je. 

\ 

88. The apparatus of claim 87, wherein the clustering technique is 
a distance metric technique. 
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89. The apparatus of claim 78^/herein the determination of the 
best representation of the current frame background is made by finding the 
background model containing the lowest percentage of foreground pixels. 
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)0. The apparatus of claim 89, wherein the foreground pixels are 
determirted using a pixel processingsystem that provides a classification 
of a pixel as either a background pixel or a foreground pixel. 



10 
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91 . A method for maintaining a background model of an image 
sequence, the^image sequence having a current frame and a previous 
frame, comprising: 

creati^ef^ maximatty-djyerse set of stored background models; 
defermiW>gfwhether a/current background model adequately 
represents an a&fcdal background of the current frame; and 

substituting grfe of the set of stored background models in 
place of the current background model if the current background model 
does not provide adequate representation. 

92. The method of claim 9^1 wherein determining whether the 
current background model provides adequate representation comprises 
finding the background model containing the lowest percentage of 
foreground pixels. 



67 



